…onents

Splits the previous catch-all ``TrainBatcher`` into four single-purpose
classes, each with explicit data-flow dependencies declared at construction
time. No more parent ``orch`` back-pointer / property-shim layer.

New module layout:

- ``batcher.py`` — ``Batcher``: pure in-memory accumulator with
  ``add / ready / pop / buffered_count``. Owns the pre-batch filters and the
  per-batch pre-filter counters. No I/O, no async, no step tracking.
- ``postprocessor.py`` — ``PostProcessor``: takes a popped batch + step
  number, runs post-batch filters, tokenizes, builds ``TrainingSample``\\ s,
  computes teacher logprobs (opd), and ships via the sender. Returns a
  ``ProcessResult`` carrying the per-rollout stats the metrics builder
  needs — so this class doesn't import pandas.
- ``eval_collector.py`` — ``EvalCollector``: buckets eval ``Trajectory``\\ s
  by ``eval_step``; flushes per-env reward / completion-len / pass@k metrics
  when the dispatcher's expected count comes back. Takes ``(expected,
  fired_envs)`` as call args instead of reaching into the dispatcher.
- ``metrics.py`` — ``MetricsBuilder``: assembles the per-step W&B dict from
  the rollout cohort + ``ProcessResult`` + dispatcher gauges. Pure-ish; only
  state is the config.

``Orchestrator`` now owns the pipeline driver loop:

- ``main_loop`` pulls trajectories off the dispatcher queue, routes train vs
  eval, drives ``batcher.add → batcher.pop`` and ``postprocessor.process``,
  builds metrics, logs to the monitor, and increments ``progress.step``.
- ``process_one_step`` is the single shipping step (ckpt save → max_steps
  check → pop → ship → barrier wait → metrics + log → step++).
- ``wait_barrier`` and ``maybe_save_ckpt`` are public helper methods on the
  orchestrator (no leading underscores; the v2 classes deliberately don't
  hide internals).

Dependency surface, post-refactor — every arrow is a real data-flow dep
declared at the call site, not a "shared context" smell:

  Batcher        ← config (batch_size / token_batch_size), pre_filters
  PostProcessor  ← config, tokenizer, renderer, mm_token_type_ids_mapping,
                   sender, teacher_inference, post_filters
  EvalCollector  ← config, monitor, eval_envs, post_filters
  MetricsBuilder ← config
  IntervalLogger ← dispatcher, policy, interval  (no more batcher refs)
  Orchestrator   ← owns all of the above

Also:

- Rename ``TrainBatcher`` → ``Batcher``: ``EvalCollector`` is the eval-side
  abstraction now, so the prefix is misleading.
- Drop the parent-pointer ``orch: Orchestrator`` constructor argument and
  the ``@property`` shim layer it forced (``dispatcher``, ``policy``,
  ``progress``, ``monitor`` etc. all read off ``self.orch`` previously).
- ``IntervalLogger`` no longer reads ``batcher.last_*`` cached state — those
  fields are gone. Step-aligned values plot fine via the per-step monitor
  log; the time-axis emit just covers dispatcher gauges + event-loop lag.
- Remove ``wikispeedia`` from ``pyproject.toml`` (stray pickup from a local
  ``uv sync --all-extras`` during dev).

Smoke: 8-step reverse-text run under v2 with the new layout, reward
0.11 → 0.24, clean event-based shutdown via ``self.stopped.set()`` at
``max_steps``.

Co-authored-by: Cursor <cursoragent@cursor.com>

…ush signals

Redesigns the data flow to be signal-based. The atomic unit emitted by the
dispatcher is now a single Rollout (one completed rollout, never a group).
Grouping moves into the sinks. Each sink owns its own flush signal:

- TrainSink groups rollouts by (env_name, example_id). When
  Rollout.is_group_complete=True arrives, the GRPO group is finalized:
  compute_advantages over the survivors, then apply_filters for the
  pre-batch pass, then drop filtered rollouts and extend batch_buf. The
  flush signal is batch_ready() — the orchestrator polls it after every
  add() and calls pop() to ship a batch.
- EvalSink groups rollouts by (env_name, eval_step). When
  Rollout.is_group_complete=True arrives, the env's eval epoch is done.
  add() returns an EvalFlush payload that the orchestrator logs.
  Filters do not apply to eval — filters are train-only by design.

Dispatcher changes:

- Trajectory (group of rollouts) replaced by Rollout (single rollout).
  maybe_finalize_group / emit_group push individual rollouts to out_q
  instead of one Trajectory per group.
- Train: the last surviving rollout of each (env, example_id) group gets
  is_group_complete=True.
- Eval: tracked per (env_name, eval_step) via eval_expected_per_env /
  eval_emitted_per_env / eval_dropped_per_env. The last expected rollout
  (emitted + dropped == expected) gets is_group_complete=True, flagging
  the env's epoch as done.
- compute_advantages is no longer called here — that's a train-sink
  concern now.
- Drops the cross-env expected_eval_counts dict and replaces it with
  per-env tracking.

Orchestrator changes:

- main_loop consumes Rollout and routes by kind. Train rollouts go to
  train_sink.add(); batch_ready() triggers process_one_step. Eval
  rollouts go to eval_sink.add(); an EvalFlush return triggers
  flush_eval (per-env metric logging on the orchestrator since it needs
  self.monitor and the logic is small).
- flush_eval is a new public method that builds per-env reward / seq-len
  / pass@k metrics from an EvalFlush and logs them via the monitor.
- post_filters is now a member field used by PostProcessor only; the
  eval path doesn't read it.

Removed:

- orchestrator_v2/batcher.py (replaced by train_sink.py)
- orchestrator_v2/eval_collector.py (replaced by eval_sink.py)

Smoke (eval.interval=5, max_steps=12, num_examples=16):

- Step 0 eval (eval_base_model) flushes cleanly: Reward 0.17.
- Step 5 eval triggers with inflight_train=121 — drain-switch overlap
  intact. Eval flushes per-env (Reward 0.54) while train continues
  (steps 6-8 ship during eval drain).
- Reward 0.14 → 0.60 over 12 steps; clean event-driven shutdown via
  self.stopped.set() at max_steps.

Both TrainSink and EvalSink now have a uniform three-level processing
structure. The dispatcher emits Rollout (atomic unit) with two boundary
signals; sinks dispatch each level to a named process function whose
logic naturally lives at that scope:

  Level    When                    Method                What it does
  -------- ----------------------- --------------------- -----------------------------
  rollout  every add()             process_rollout       train: backfill + interleave (eager tokenization)
                                                         eval:  no-op
  group    is_group_done=True      process_group         train: compute_advantages + pre_batch_filters
                                                         eval:  move per-example group into env batch bucket
  batch    batch_ready (train)     process_batch         train: post_batch_filters + assemble samples + metrics
           is_batch_done (eval)                          eval:  build per-env metrics (reward, pass@k, ...)

Dispatcher Rollout dataclass:

- is_group_complete split into two boundary signals:
  - is_group_done: last rollout of the (env, example_id) group. Set for
    both train (GRPO group) and eval (per-example group).
  - is_batch_done: last rollout of an eval env-epoch (emitted+dropped ==
    expected). Always False for train — the train sink decides batch
    boundaries itself from batch_size / token_batch_size.
- emit_group sets both flags appropriately per kind.

TrainSink:

- async add(rollout) returns True when the batch is ready to pop.
- process_rollout is the eager-tokenize hook. Per-rollout tokenization
  overlaps with the dispatcher producing more rollouts, removing the
  parallel-tokenize-all-at-ship-time sync point.
- process_group runs compute_advantages, propagates advantage / reward /
  env_name onto the already-tokenized TrainingSamples, then runs the
  pre-filter pass.
- pop_batch(step, progress, gauges, ...) runs process_batch internally
  and returns a TrainBatchResult { rollouts, samples, metrics, result }.
- Owns the MetricsBuilder reference; per-batch metrics are part of
  process_batch, not a separate orchestrator step.

EvalSink:

- add(rollout) returns an EvalBatchResult when an env-epoch is complete
  (is_batch_done arrived).
- process_rollout is a no-op (kept for structural symmetry with
  TrainSink).
- process_group buckets the per-example group into the (env, eval_step)
  batch.
- process_batch / build_metrics produces the eval/{env}/ metrics dict
  (reward, pass@k, etc) — no filter calls; filters are train-only.

Orchestrator:

- main_loop is a thin router: await train_sink.add(...) — if True, run
  process_one_step; eval_sink.add(...) — if EvalBatchResult, run
  log_eval_batch.
- process_one_step is now I/O-only: save_rollouts, offload_images,
  teacher_logprobs (opd), wait_barrier, sender.send, monitor.log,
  heartbeat, ckpt save, step++. All processing already happened in the
  sink.
- log_eval_batch is a small side-effect method: save_rollouts +
  monitor.log_eval_samples + monitor.log; metrics dict comes pre-built
  from EvalSink.
- postprocessor.py is gone — its work split across TrainSink (filter +
  tokenize + samples + metrics) and orchestrator (ship I/O + opd
  logprobs + disk side effects).
- ProcessResult moved into metrics.py to avoid a circular import.

Smoke (eval.interval=5, max_steps=12, num_examples=16):

- Step 0 eval flushes cleanly: Reward 0.14.
- Step 5 eval triggers with inflight_train=115 — drain-switch overlap
  intact. Train steps 6/7/8 ship during eval drain. Eval flushes
  per-env: Reward 0.48.
- Reward 0.13 → 0.64 over 12 steps; clean event-driven shutdown.

…nd eval

Both sinks now use the same flag — ``Rollout.is_batch_done`` — to signal the
batch boundary. The orchestrator's main loop polls a single, uniform
condition; the only difference is who stamps the flag:

- Eval: the dispatcher stamps ``is_batch_done=True`` on the last expected
  rollout of an env-epoch (emitted+dropped == expected). No change here.
- Train: ``TrainSink.add`` now stamps ``rollout.is_batch_done = True`` after
  ``process_group`` if its buffer has reached ``batch_size`` / token
  budget. The polling counterpart is ``TrainSink.is_batch_done()`` (renamed
  from ``batch_ready()`` for naming symmetry with the rollout flag).

Orchestrator main loop:

  await self.train_sink.add(rollout)
  while rollout.is_batch_done and not self.stopped.is_set():
      await self.process_one_step()
      rollout.is_batch_done = self.train_sink.is_batch_done()

Same predicate (``rollout.is_batch_done``) drives both sinks. The eval
path consumes the flag inside ``eval_sink.add``; the train path consumes
it in the main loop's ``while``.

Smoke: reverse-text 8-step run under v2, reward 0.13 → 0.36, clean
event-driven shutdown via ``self.stopped.set()`` at max_steps.

…wcase

- Drop ``configs/debug/orch_v2.toml``. v2 is opt-in via a CLI flag —
  ``--orchestrator.experimental.use-orch-v2`` (or the equivalent TOML
  key) — and a one-liner overlay TOML was overkill for that.
- Add ``configs/hendrycks_math/orch_v2_demo.toml`` — a small (2 GPU,
  30 train step) eval-heavy run that makes the v2 overlap win visible
  on wall clock. 6 AIME2024 eval epochs of 30 × 4 = 120 long-reasoning
  rollouts each; under the legacy orchestrator each epoch pauses train,
  under v2 they overlap. Comment block at the top of the file shows
  how to run both side-by-side via the CLI flag.

The eval boundary is a strict function of ``progress.step``, ``eval.interval``,
``eval_base_model``, and ``skip_eval_on_resume`` — no extra persistent state
needed. ``progress.last_eval_step`` was only ever written (in
``log_eval_batch``) and never read; the dispatcher's own
``last_eval_step_per_env`` dict handles "don't re-trigger at the resumed
step" and is seeded from ``resume_step`` directly.

The ``Progress`` dataclass on the ckpt now carries just ``step`` + totals.
Old checkpoints containing the extra ``last_eval_step`` field load fine
— the ``hasattr(progress, key)`` guard in ``CheckpointManager.load`` skips
unknown serialized fields silently.

…ypes

Rename `EvalBatchResult` → `EvalBatch`, `TrainBatchResult` → `TrainBatch`,
and rename `eval_step` → `step` on the eval payload. The dataclasses now
carry only the raw rollouts + per-rollout counters; the `metrics` dict is
a pure view computed at log time.

Why: the `metrics` field duplicated `rollouts` + a few scalars and forced
the orchestrator to build with a `step_time=0.0` placeholder inside
`TrainSink.pop_batch` and mutate the dict after the post-barrier wait.
Building at log time with the real timings keeps one source of truth.

- Drop `metrics` from both batch payloads; orchestrator calls
  `MetricsBuilder.build` (train) and `EvalSink.build_metrics` (eval) when
  ready to log.
- Drop `parallel_preprocess_time` / `teacher_logprobs_time` from
  `ProcessResult` (never read in v2 — tokenization is per-rollout;
  teacher logprobs is an orchestrator scalar passed to `build` directly).
- `TrainSink` no longer holds a `MetricsBuilder` ref; single-purpose
  ingest + payload assembly.

Co-authored-by: Cursor <cursoragent@cursor.com>

Move every dataclass / type alias / protocol from the orchestrator_v2
modules into a single ``types.py``. Behavioral modules (dispatcher, sinks,
watcher, log_loop, ckpt, metrics, orchestrator) now import their payloads
from one place instead of from each other.

Moved:
- ``Policy`` (was policy.py — file deleted, only contained the dataclass)
- ``Progress`` (was ckpt.py)
- ``Kind``, ``SchedMode``, ``Rollout``, ``RolloutMeta``, ``GroupState``
  (were dispatcher.py)
- ``EvalBatch`` (was eval_sink.py)
- ``TrainBatch`` (was train_sink.py)
- ``ProcessResult`` (was metrics.py)
- ``VersionObserver`` (was watcher.py)

Co-authored-by: Cursor <cursoragent@cursor.com>

Let train/eval rollouts use the full 8k max_model_len so the demo
showcases the overlap on realistic long-reasoning AIME completions
instead of capping them at 4k.

Co-authored-by: Cursor <cursoragent@cursor.com>

… sink

Previously the dispatcher stamped ``is_group_done`` / ``is_batch_done``
flags on whichever rollout it happened to iterate last in ``emit_group``.
But "last in a group" is whichever straggler finishes last — not knowable
at dispatch time — so the flag-on-rollout model was always derivative of
arrival ordering rather than a real signal.

Flip the design so the dispatcher emits every dispatched rollout (success,
env-error, empty-trajectory, task-exception, off-policy-cancel) exactly
once to ``out_q`` and the sinks own group/batch finalization by counting:

- Drop ``is_group_done`` / ``is_batch_done`` from ``Rollout``.
- ``TrainSink.add`` finalizes a group on the ``group_size``-th arrival,
  returns ``bool`` for batch-ready (was: mutate ``rollout.is_batch_done``).
- ``EvalSink.add`` finalizes a group on the ``group_size``-th arrival,
  finalizes the epoch when total arrivals reach
  ``num_examples * group_size``, returns ``EvalBatch | None``.
- Sinks decide drop / partial-train policy: filter errored rollouts before
  ``compute_advantages``; drop the whole group when the env
  ``requires_group_scoring`` and any rollout failed.
- Dispatcher synthesizes ``vf.RolloutOutput`` markers with ``error`` set
  for task exceptions and off-policy cancellations, so the sink's count-
  based finalization still fires when rollouts never produced real output.
- Drop ``eval_expected_per_env`` / ``eval_emitted_per_env`` /
  ``eval_dropped_per_env`` side-channel bookkeeping — pure dead weight
  once the sink counts arrivals itself.
- Drop ``GroupState.completed_rollouts`` / ``failed_rollouts`` — the
  dispatcher no longer accumulates rollouts before emit; ``emitted``
  counter is enough to pop the group from ``self.groups`` once every
  member has flowed out.

Co-authored-by: Cursor <cursoragent@cursor.com>

… check

The single-node SLURM launcher dumps the fully-resolved ``RLConfig`` to
disk and re-loads it inside the SLURM job. The dumped TOML has both the
shared-level keys (``output_dir``, ``model.name``, ``wandb.*``, ...) and
the matching sub-config keys (filled by the propagator on the original
load) populated to the same value. The old conflict check flagged
``shared is set, sub is also set`` regardless of values, so the re-load
failed with a ``Shared config conflicts with matching sub-config field(s)``
error.

Tighten the check to "shared and sub disagree" — matching values are a
harmless roundtrip (the propagator's fill-if-absent would have produced
the same thing). Genuine conflicts (e.g. shared ``model.name = "A"`` vs
``trainer.model.name = "B"``) still raise.

Covers the three sites that build ``conflicts``: the ``propagate``
helper, the ``output_dir`` block (matches against the per-target expected
value, including ``orchestrator.output_dir = f"{output_dir}/run_default"``),
and the ``wandb.name`` block (matches against the ``-trainer`` /
``-orchestrator`` suffixed form in non-shared mode).

Co-authored-by: Cursor <cursoragent@cursor.com>

…ation

Promote the v2 design to be the orchestrator. Clean deprecation — no
``use_orch_v2`` flag, no parallel ``orchestrator-v2`` console script, no
``prime_rl.orchestrator_v2`` module. The v2 sources move into
``prime_rl.orchestrator`` and the legacy files (``orchestrator.py``,
``scheduler.py``, ``buffer.py``, ``ckpt.py``) are deleted outright.

The legacy ``orchestrator`` console script keeps working — it now points
at the new ``Orchestrator(config).start()`` class. Existing TOML configs
work as-is (the public surface didn't move).

To compare against pre-v2 numbers, check out ``main`` and re-run any of
the existing configs head-to-head.

Co-authored-by: Cursor <cursoragent@cursor.com>

Roll back the propagator conflict-check loosening from this branch. The
single-node SLURM dump-then-reload bug it was working around is a
pre-existing issue on the launcher path; will be addressed in a separate
PR rather than ride along with this orchestrator rewrite.

Local launcher (the default ``rl_local`` path) is unaffected by the
validator either way.

Co-authored-by: Cursor <cursoragent@cursor.com>

…cher

Mirror the ``Train*`` / ``Eval*`` naming used everywhere else in the
orchestrator (``TrainEnvs`` / ``EvalEnvs``, ``TrainSink`` / ``EvalSink``,
``TrainBatch`` / ``EvalBatch``). Both abstractions answer the same
question — "what's the next example to schedule?" — with the asymmetry
their roles demand:

- ``TrainSource`` (was inline ``TrainEnvCycle``) — infinite pull. Same
  weighted round-robin over training envs as before; just renamed and
  moved into its own module.

- ``EvalSource`` (new) — trigger-driven finite-per-epoch pull. Owns the
  per-env example lists, per-env intervals, the per-(env, step) FIFO
  queue, ``last_eval_step_per_env``, ``eval_base_model`` startup
  handling, and ``skip_eval_on_resume`` gating. Exposes
  ``trigger(step)`` / ``trigger_at_start()`` for the dispatcher to poke,
  and ``peek`` / ``pop`` / ``bool(source)`` / ``len(source)`` for it to
  pull.

Dispatcher net loss: ~150 lines and 5 fields (``eval_queue``,
``eval_examples``, ``eval_intervals``, ``eval_step_envs``,
``eval_at_zero_pending``, ``last_eval_step_per_env``, ``resume_step``)
plus two methods (``fire_eval_epoch``, ``enqueue_eval_env``).
``RolloutDispatcher.on_new_version`` is now a 3-line delegation:
``fired = eval_source.trigger(step) ; if fired: switch_mode + log``.

Also drops ``eval_step_envs`` entirely — pure dead state: only ever
written, never read.

Co-authored-by: Cursor <cursoragent@cursor.com>

…tch, add eval cancellation

A bundle of orchestrator/dispatcher cleanups all sitting in this PR:

**dispatcher → just a scheduler**
- ``__init__`` takes explicit params (``train_source``, ``eval_source``,
  ``inference``, ``policy``, ``max_inflight_rollouts``,
  ``tasks_per_minute``, ``group_size``, ``max_off_policy_steps_train/eval``,
  ``training_mode``) instead of the full ``OrchestratorConfig``. Sources
  are now orchestrator-owned and passed in.
- Drops ``resume_step`` — it's purely an ``EvalSource`` concern now and
  the orchestrator passes it directly to ``EvalSource``.
- ``DispatcherMetrics`` dataclass on ``types.py`` consolidates all the
  per-poll counters (cancellation by kind, error/empty by env,
  errors-by-type, total arrivals, eval_epochs_started, mode_transitions)
  with ``record_*`` helpers and ``drained()`` that returns the wandb
  shape + clears the drain counters in one shot. Replaces 6 dict/int
  fields previously scattered on the dispatcher.
- Drops the redundant ``inflight_total_permits`` property (was just an
  alias for ``inflight_permits``).
- Unifies ``try_schedule_train`` + ``try_schedule_eval`` into a single
  ``try_schedule(kind)`` + ``next_fresh_group(kind, envs)``. Same
  "continue existing group → otherwise open a fresh one" algorithm for
  both; the only asymmetry (train pulls from an infinite source vs. eval
  peeks+pops a finite queue) sits in two clearly-labeled branches of
  ``next_fresh_group``.

**eval triggers move to per-batch (orchestrator-side)**
- ``Orchestrator.process_one_step`` calls ``maybe_trigger_eval(step)``
  at the end (after ``progress.step += 1``); ``Orchestrator.start``
  calls ``maybe_trigger_eval(at_start=True)`` once before the main loop.
- ``RolloutDispatcher.on_new_version`` now does only the off-policy
  bookkeeping (bump counters, cancel stale groups). Eval epoch
  triggering is no longer tied to "we just got new weights"; it's tied
  to "we just completed a training step", which matches the conceptual
  model.

**eval cancellation + cancelled-as-errored**
- ``max_off_policy_steps_eval: int | None = None`` (default uncapped —
  preserves current behavior) on ``OrchestratorConfig``. When set, eval
  rollouts older than the cap get cancelled via the same drain-and-
  switch path as train (synthetic ``Cancelled`` rollouts flow to the
  eval sink). Train and eval caps can now diverge.
- ``EvalSink.build_metrics`` filters errored rollouts out of all stats
  (reward/seq_len/pass@k) and reports them as ``cancelled_count`` /
  ``errored_count`` / ``valid_rate`` separately. Counting failed
  attempts as reward=0 would silently bias scores down.

**``eval.eval_base_model`` renamed to ``eval.skip_first_step``**
- New default is False (no step-0 eval — opt in instead of opt out).
  The legacy ``eval_base_model`` key still parses via a silent
  ``AliasChoices`` (same boolean semantics), so configs that explicitly
  set ``eval_base_model=true|false`` keep working. Configs that relied
  on the implicit default get the new no-step-0-eval behavior.

**misc**
- Drop ``OrchestratorConfig.seed`` (hardcoded ``seed=42`` for
  ``TrainSource`` in the orchestrator — was effectively the default
  anyway).
- Drop ``eval_step_envs`` dead state (write-only).
- Drop ``record_dropped_group`` from ``DispatcherMetrics`` — group
  dropping is a sink concern now (see prior commit).

Co-authored-by: Cursor <cursoragent@cursor.com>

…off_policy_steps

Drop the global ``OrchestratorConfig.max_off_policy_steps_eval`` field and
instead make off-policy caps a per-side override, matching the same
shared-default / per-side-override pattern used everywhere else in the
config (e.g. ``train.sampling`` vs per-env sampling):

- ``OrchestratorConfig.max_off_policy_steps`` (already existed, default
  8): the shared default for both train and eval.
- ``TrainConfig.max_off_policy_steps: int | None = None``: per-train
  override; ``None`` inherits the global.
- ``EvalConfig.max_off_policy_steps: int | None = None``: per-eval
  override; ``None`` inherits the global.

The orchestrator resolves both sides to plain ints before passing to the
dispatcher, so the dispatcher's signature tightens from ``int | None`` to
``int`` for ``max_off_policy_steps_eval`` (caps are never "unset" by the
time they reach the dispatcher — they either come from the per-side field
or fall back to the global default).

Behavior change worth flagging: the previous ``max_off_policy_steps_eval
| None = None`` defaulted to *uncapped* (eval was never cancelled). The
new pattern defaults eval to the same cap as train (``8`` by default),
so eval rollouts that lag past the cap will now be cancelled by default.
Set ``eval.max_off_policy_steps`` to a large value if the previous
uncapped behavior is preferred.

…_for→batch_size_for

Two follow-ups from review that didn't ride along the previous commit:

- ``EvalSink.process_batch`` now builds the per-env metrics dict inline
  and stamps it on the returned ``EvalBatch.metrics`` field. Was an
  awkward two-step (``process_batch`` returned just rollouts, then the
  orchestrator separately called ``build_metrics(batch)``); per-batch
  derivations belong in the per-batch hook. ``build_metrics`` as a
  standalone method goes away.

- ``EvalSource.expected_for`` → ``batch_size_for`` (it's the same number
  the train sink calls ``batch_size`` — total rollouts per epoch).

…cy_steps

Roll back ``TrainConfig.max_off_policy_steps`` / ``EvalConfig.max_off_policy_steps``.
Keep just the global ``OrchestratorConfig.max_off_policy_steps``; it applies
to train and eval rollouts the same way. Eval cancellation behavior
(introduced earlier in this branch — cancelled rollouts surface as
``Cancelled`` markers, sink filters them out of stats) stays in place;
this commit only reverts the per-side override scaffolding.

# Conflicts:
#	src/prime_rl/orchestrator/orchestrator.py

The ``SharedWandbConfig.deprecate_shared_field`` ``model_validator(mode="before")``
introduced in #2649 popped ``wandb.shared`` from input dicts with a
``FutureWarning`` so older configs that still set ``shared = true`` (or
``false``) kept loading.

The ``rl`` entrypoint always uses shared W&B mode now, the legacy split with
``-trainer`` / ``-orchestrator`` suffixes is gone, and the in-tree configs no
longer set ``shared``. Drop the shim so configs that still carry the field
fail loudly via the default ``extra = "forbid"`` instead of silently warning.

Docs and CHANGELOG updated.

Co-authored-by: Cursor <cursoragent@cursor.com>

pydantic-config's CLI negation (``--no-wandb`` / ``--no-ckpt``) writes the
string ``"None"`` into the override dict, which only becomes real ``None``
later via ``BaseConfig._none_str_to_none``. ``RLConfig``'s subclass
``mode="before"`` validators run *before* that conversion, so the bare-block
enablement in ``propagate_shared_fields`` used to see the string and
re-enable ``trainer.wandb`` / ``orchestrator.wandb`` (or the ``ckpt`` pair)
with defaults — silently undoing the CLI's intent.

Tighten the bare-block check from ``get(key) is not None`` to
``isinstance(get(key), dict)``. A bare ``[wandb]`` block in TOML is ``{}``
(dict, still enables); ``--no-wandb`` produces ``"None"`` (string, now
correctly skipped). Same logic applies to ``ckpt``.

Verified end-to-end with ``cli(RLConfig, args=[..., "--no-wandb"])``:
``cfg.wandb``, ``cfg.trainer.wandb``, ``cfg.orchestrator.wandb`` are all
``None``. ``--wandb None`` and ``--no-ckpt`` behave the same. Bare
``[wandb] {}`` / ``[ckpt] {}`` blocks still enable the sub-configs.

Co-authored-by: Cursor <cursoragent@cursor.com>

It's a bug fix, not a breaking config change, so it doesn't belong in
CHANGELOG.md (which documents breaking config changes only).

Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Cursor <cursoragent@cursor.com>

``docs/`` should describe the current state of the codebase, not its
history. Strip the "the legacy split has been removed" sentence from
``docs/training.md`` — readers on ``main`` don't care that there used to
be a non-shared mode — and add a ``## Docs`` rule to ``AGENTS.md`` to
codify it.

Co-authored-by: Cursor <cursoragent@cursor.com>

dp in multi-node inference is per-node (data_parallel_size_local),
multiplied by num_infer_nodes internally. dp=16 double-counted →
inference_world_size=32. dp=8 × 2 infer nodes = 16 global ranks.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

num_infer_nodes=2 builds ONE inference instance spanning 2 nodes
(cross-node DP coordinator → slow/fragile rendezvous). For a 4B model
on 1 GPU we want 2 independent single-node replicas behind the router:
num_infer_nodes=1 + num_infer_replicas=2 (dp=8/node → 16 infer GPUs).
Same 2x throughput, no cross-node coordination.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

start()'s teardown now logs a forced-cleanup warning when main_loop is
interrupted (signal CancelledError / KeyboardInterrupt / error) instead
of the clean-exit success line, so an interrupted run is distinguishable
from a true completion.

print_benchmark was unused in the orchestrator package (the trainers use
the copy in trainer/utils.py); drop it and its now-orphaned pandas / rich
/ format_time / Any imports.

Finish moving the color-codeword multimodal configs out of the
configs/multimodal/ dir: drop the unused rl_color_codeword.toml /
rl_color_codeword_test.toml and trim the renamed
configs/debug/multimodal.toml header. The dir is now empty.

Eval rollouts were generated via select_train_client (the renderer/token
train client) instead of get_eval_client (chat-completions). That changed
the eval generation path vs the legacy orchestrator (which used the chat
eval client), producing a consistent eval-score offset. Route eval groups
through pool.get_eval_client() so eval matches the legacy path; train
still uses the renderer/token train client.

Set [orchestrator.renderer] name="qwen3.5", enable_thinking=false
(Qwen35RendererConfig field) — text-only SWE, no <think> traces. Was
relying on the auto renderer (default thinking on for 4B).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The on-main config is the non-thinking version (enable_thinking=false)
+ prefix caching + language_model_only. Run length 100 -> 400.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>